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Foreword 



In November, 1986, the Colorado Commission on Higher Education issued a request 
for proposal for the development of a white paper on educational 
accountability. The scope of the paper requested by this RFP was broad, 
covering: 

• policy issues related to state and institutional effectiveness 

• relationships of master plans to system effectiveness 

e the significance of role, mission, and scope statements to the 
evaluation of institutional effectiveness 

• application of the principles discussed in the siate of Colorado 

• a proposal for action at state and institutional levels 

• criteria to evaluate institutional and system performance. 

In late December, a contract was awarded to the National Center for Higher 
Education Management Systems to prepare this paper. 

During the month of January, 1987, the paper passed through four developmental 
stages—a preliminary outline, a detailed outline, a first draft document, and 
the final product. At each stage, NCHEHS staff involved in the project met 
with Commission staff to review the most recent material. Throughout this 
process the Commission staff members were most helpful, providing conscientious 
review and substantive comments on our draft materials, helping us to 
understand the Colorado context into which we were placing the concepts with 
which we were working, and ensuring that we promptly received all the 
background materials we felt we needed to complete this task. 

While the staff members were most helpful at each step of the process, they 
gave us complete freedom to shape the contents of this paper as we deemed most 
appropriate. We want to thank Blenda Wilson, Charles Manning, Frank Armijo, 
Mark Chisholm, and Martha Romero for their assistance and to absolve them of 
any responsibility for errors in fact or logic that appear in this document. 
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Executive Summary 



This white paper on educational accountability deals with three distinct 
topics. First, the concept of educational accountability is developed and 
related to other important processes such as master planning, role and mission 
definition, program review, and resource allocation. Second, these concepts 
are utilized as templates for analyzing current practice in the state of 
Colorado and for discussing the set of expectations incorporated in the various 
provisions of HB1187. Third, a set of recommendations are presented for action 
at both state and institutional levels. 

In developing the concept of accountability, the central themes that emerge are 
as follows: 

• Being accountable includes both effectively discharging an obligation 
and being answerable in thiat regard. Thus, the concept of 
accountability embraces issues of both performance and communication , 

• These issues revolve around the questions of 
To whom? 

Are what obligations owed? 
By whom? 

• For purposes of this paper, the "to whom?" is presumed to be students 
and their parents, the public, and the Governor and General Assembly. 

• Specific — and different — accountability obligations are owed by 
institutions and by the state higher education agency. A discussion of 
the distinctions between system-level and institutional- level 
accountability is a major feature of this paper. 

• The nature of the obligation owed can be expressed in terms of 

- student outcomes 

- contributions to state priorities 

- maintenance and improvements to the educational infrastructure within 
the state — the system of higher education and the individual 
institutions that comprise that system. 

Following a detailed discussion of the concept of educational accountability, 
we explain the relationships between accountability and several of the * 
Commission's other functions. The focus of this discussion is on the linkages 
between accountability and 

• Master planning and the specification of state priorities 

• Institutional role and mission and the shape of the educational system 
of the state 

• Program review and both the achievement of state priorities and the 
refinement of institutional role and mission 

• Resource allocation and the provision of incentives to accomplish state 
priorities 
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Having dealt with these topics at a conceptual level, we proceed to a 
discussion and analysis of the various substantive provisions of HB1187, using 
the concepts we have developed. The purposes are to place them within a single 
framework and, by doing so, show l) the topics to which they speak and the 
topics on which they are silent and 2) their interrelatedness. 

Finally, we offer a series of recocinendations for both state-level and 
institutional action. These recommendations fall into four broad categories: 

• Recommendations to CCHE with regard to system-level accountability 

i Recommendations to CCHE regarding institutional action on selected 
statewide priorities 

• Recommendations to CCHE regarding institutional accountability 
processes 

• Recommendations concerning institution-level action. 

The reader is referred to Section V of the paper for the substance of these 
recommendations. 
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Accountability in Higher Education: 
Meaning and Methods 



I. BACKGROUND 

H0U-.8 Bill No. 1187, enacted in 1935, reestablished the Colorado Commission on 
Higher Education (CCHE) and delegated to it a variety of powers and duties. 
Article 13 of that bill mandates implementation of a higher education 
accountability program and directs CCHE to develop policies under which various 
governing boards and institutions are to carry out its provisions. In this 
Article, legislators specified many ".ground rules" for such an accountability 
program— for example, "institutions shall design systematic programs to assess 
Imowledge, capacities, and skills developed by students," and "the results of 
the assessment shall be communicated to the public" — but delegated the 
formulation of specific policies governing the program to the Commission. 

As an initial step in formulating policy, CCHE issued a request for proposal to 
develop a white paper on accountability. This document was intended to serve 
as the focal point for "a dialogue with the higher education community, the 
General Assembly, and the public as to the appropriateness of various policy 
options." The National Center for Higher Education Management Systems 
(NCHEHS), a nonprofit organization with offices in Boulder, Colorado, was 
selected to prepare the paper. 

The contents of the paper have been shaped by several global considerations. 
First, the term "accountability" has no commonly understood, operational 
meaning when applied to the performance of higher education institutions. 
Although the term is widely used and is generally understood to indicate the 
condition of being held responsible for meeting established obligations, 
definitions have not progressed beyond this simple formulation. As a 
consequence, we considered it imperative that the paper attempt to establish a 
common definitional base from which subsequent dialogue about accountability 
can proceed. Second, the mantle of "obligation" falls not only on institutions 
and their associated governing boards, but on CCHE aS well. As a result, we 
considered that the concept of higher education accountability must be 
developed in such a way that the obligations and responsibilities of both 
institutions and state agency are related and clarified. Third, HB1187 assigns 
CCHE numerous duties in addition to accountability. Included, for example, are 
mandates to develop a master plan for postsecondary education in Colorado; to 
ensure that functioning program review processes are developed and maintained 
by each institution; to approve all new programs and establish criteria for 
program discontinuance; and to further define the role and mission of each 
Colorado institution of higher education. While presented in the legislation 
as unrelated items, these responsibilities are, in fact, highly interdependent. 
We felt that the more clearly the links between accountability and other 
functions were explained, the more useful the paper will be. Finally, the 
general phrase, "higher education accountability," takes on specific meaning in 
the context of the statutes, history, and current events of the state of 
Colorado. To make the paper more useful to the intended audiences, 
responsibilities of CCHE for a variety c£ functions are discussed and 
explicitly related to accountability. 

With these considerations in mind, wp have prepared a paper comprised of four 
major sections. Section II explores the notion of accountability in some depth 
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and develops a conceptual framework designed to highlight initial differences 
between state (system) level accountability and institution-level 
accountability; it also covers the factors that make accountability 
requirements different for institutions with different missions. Section III 
discusses relationships between accountability and other major functions of 
state nigher education agencies, for example, master planning, finance, and 
program approval/review. Section IV particularizes the contents of Sections I 
and III to the state of Colorado, discussing the priorities and requirements 
that emerge from HB1187 and other sources in a Colorado context. Finally, 
Section V contains our observations and recommendations for "next steps" for 
CCHE consideration and action. Finally, we have included in an appendix some 
alternative approaches to assessing performance for both institutions and the 
state as a whole. We have also included in an appendix a list of reference 
works we believe will be helpful as the Commission moves to its next steps. 
Throughout, we have attempted to render the discussion complete and concrete 
without being unnecessarily burdensome. As always, saying enough without 
saying too much represents a significant challenge. He can but hope that we 
have succeeded . 
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THE CONCEPT OF ACCOUNTABILITY 



A. Introduction 

A first principle of effective communication is that the parties to any 
discourse must have a common language. Because the torm "accountability" 
is a core concept, it is important that it be similarly understood by all 
parties at interest. These include institutions and governing boards, the 
General Assembly, the public at large, and CCHE itself. Unfortunately, 
providing a definition is not a simple task. One dictionary (the World 
Book Dicti onary , 1982) defines accountability as "the state of being held 
responsible for carrying out one's obligations." The adjective, 
"accountable," is similarly defined as "answerable." Inspecting these 
simple definitions, two fundamental aspects of accountability emerge— (l) 
specifying a domain of obligation, and (2) being answerable for the 
effective discharge of that obligation. The first focuses on defining 
responsibilities or performance; the second focuses on communicating 
evidence of specified performance to audiences deemed entitled to it. 
Furthermore, inspection of this definition leads to a core question that 
frames any particular instance of "accountability." This question can be 
trisected as follows: 



• To whom? 

• Are what obligations owed ? 

• By whom? 

A brief discussion addressing each of these points will pave the way for a 
more in-depth treatment of the topic. 

First, to whom is accountability owed? Higher education is a pluralistic 
enterprise serving a multitude of constituents. Among the most obvious 
are students (current and prospective) and their parents, executive and 
legislative branches of state government, and the public. Others, perhaps 
less salient, that must nevertheless be considered are employers, 
accrediting bodies, and other funders including the federal government and 
philanthropic organizations. For purposes of this paper, we will give 
primary attention to the ^rior set. 

Second, what obligations are owed? Higher education is a multi-faceted 
enterprise, and there are thus many potential dimensions of obligation. 
In the recent past, questions of oblig^ition/rasponsibility were framed 
primarily in terms of cost efficiency and in terms of adherence to 
established procedures in such eureas as accounting, purchasing, and 
hiring. The emphasis was on the means of education. In response, higher 
education institutions and agencies developed numerous, and generally 
adequate, mechanisms for providing evidence of responsible behavior. 
Recently, however, the emphasis of accountability has changed. Now the 
major accountability issues are those that surround the ends of education. 
Primary emphasis is placed on what is actually produced, on effectiveness 
rather than efficiency. This change is profo!Jtr^, and our capacities to 
respond to it are generally inadequate. Indee , many still proclaim the 
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practical folly of any such endeavor. But lack of a fully developed 
technology for assessing outcomes should not prevent us from stating as a 
basis for obligation the kinds of outcomes that ought to be produced. 
While this paper covers both efficiency and effectiveness in considering 
accountability, it is largely oriented toward the latter consideration. 

Finally, who is to be held accountable? "Higher education" is far from a 
monolithic enterprise, and responsibility is widely distributed* Certain 
types of responsibility reside at the level of the institution itself, and 
at the levels of subunits such as schools, colleges, and programs within 
the institution. Other types of responsibility reside at the level of the 
individual administrator, faculty member, and student. Finally, a special 
responsibility falls on those who formulate policy, allocate resources and 
monitor the performance of the entire system of postsecondary education 
within a state.* Forced to make a simplifying choice among this array, we 
deal only with the concept of accountability as it pertains to 
institutions and to CCHE — the agency having some responsibility for the 
system of higher education in the state of Colorado. 

In Slim, we have limited discussion to accountability that is l) owed to 
state government, students, and the public, 2) primarily for matters of 
institutional effectiveness, 3) by institutions and the state higher 
education agency. This domain is illustrated by figure 1 . 



B. The Substance of Accountability 

Of the three component elements of "accountability" identified above, it 
is in the second that the basic conceptiial issues lie. While questions of 
by whom and to whom obligations are owed still spark debate, discussion is 
far more frequently centered on the nature of the obligation rather than 
on its existence . Part of the issue is undoubtedly rooted in legitimate 
differences about the boundaries of authority among various parties at 
interest. We contend, however, that the absence of a soiond conceptual 
framework for describing the substance of obligation is the primary cause 
of higher education's inability to deal rationally with the topic. 

Most would agree that the obligation owed by institutions and agencies to 
their respective constituents is a demonstration of their effectiveness 
and their efficiency, and the rhetoric of higher education is replete with 
reference to these twin objectives. But what do we really mean by these 
terms? As used in the balance of this paper, the term effectiveness 
refers to the extent to which institutions and agencies of higher 
education accomplish their intended purposes or achieved desired ends. 
The term efficiency refers to the ability of institutions and agencies to 
accomplish desired ends while minimizing the use of resources. 
Unfortunately, historical difficulties of defining and assessing 



♦Here the term "system" is used in an informal rather than the formal 
(governance) sense and includes all institutions in the state (public, private, 
and proprietary, as well as any out-of-state institutions offering programs in 
the state) that, collectively, contribute to achieving desired state-level 
objectives. 
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Figure 1 
The Domain of Accountability 
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effectiveness have too often led to a condition in which efficiency 
(translated as cost minimization) becomes an end in itself. This 
imbalance must be redressed; a necessary first step is to propose an 
understandable, commonly agreed-upon way of describing effectiveness. 

As stated above, we propose that effectiveness first be defined as 
achieving intended outcomes. This deceptively simple definition hidec a 
series of fundamental issues. First, in using the word, outcomes, we 
presume to know what educational outcomes, in fact, are. In using the 
word, intended , we raise the corollary question of "whose intention?" 
Finally, in using the word, achieving , we imply a capacity to assess or 
measure. Some of the measurement issues are addressed in an appendix to 
this report. The remainder of this section presents a method for defining 
"intended outcomes" in operational terms. 

In beginning this task, it is most useful to first conceive of higher 
education as an integrated system of entities — institutions, agencies, and 
individuals interacting ir» pursuit of their own ends — and to recognize 
outcomes as consisting oJ' changes (or maintenance) in state or condition 
within the entities of the system. From the point of view of each actor 
or entity within the system, outcomes can be expressed as: 

a) Changes induced in other entities within the system 

b) Conditions or improvements made internally 

Outcomes can thus be expressed both in terms of what is done to others as 
a result of system operations, and in terms of assets, tangible and 
intangible, acquired by each actor. For example, the knowledge or skills 
imparted by a college to a student represent the first kind of outcome, a 
change induced by the institution in another entity, the student. 
Creation of a high quality faculty or a well-designed curriculum 
represents the second kind of outcome, a change in capacity or asset 
within the institution itself. This classification of desirable outcomes 
reflects the nature of the managerial task that all policymakers confront. 
All must be as concerned about creating and maintaining the productive 
capacity or "infrastructure" of their enterprise as they are with ensuring 
that it in fact produces something of value. 

This construction highlights the need to focus on both the entities being 
held accountable and on the entities that stand to gain or lose* For 
purposes of this paper, we have already identified the accoiontable parties 
as public institutions and the state higher education agency (CCHE). We 
will now identify the prime beneficiaries of the system' s operation as: 

• The state as a whole (its economy, literacy of the population, 
etc.) 

• The state system of postsecondary education 

• Individual institutions 

• Individual programs within institutions 
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Figure 2 

The Entities in the System of Higher Education 




• Individual students 

In assessing effectiveness in terms of changes in the state system, 
individual institutions, and programs, the question is primarily one of 
"infrastructure"— the nature, quality, capacity and appropriateness of 
educational assets. Results of applying these assets are found in 
desirable changes of condition in the state as a whole, in identified 
groups of students, or in individual students. 

Having identified the primary units for analysis, a central question 
remains: What specific conditions or changes for each entity should be 
included within the purview of educational accountability? In essence, 
this question asks us to identify the particular outcomes of importance 
for each entity in the system. In providing em initial answer, we do not 
intend to present an academic treatise on outcomes. Rather we wish to 
initially map the terrain, emd to propose characteristics of each entity 
that might appropriately be considered within the domain of 
accountability. We present these characteristios in schematic form below: 

1 . Dimensions of Student Outcomes 

For individual students, the characteristics of interest are as 
follows: 

• knowledge outcomes , including 

- general knowledge, the kind of knowledge associated with 
broad fields of study and different modes of inquiry and 
investigation in the sciences, social sciences, humanities, 
and the arts. 

- specific knowledge, the kind of "major field" knowledge 
associated with specific academic disciplines. 

• skills outcomes , including 

- general skills, such as reading, writing, speaking, and 
computing 

- "higher order" skills, such as analysis, synthesis, critical 
thinking, and problem-solving 

- vocational or professional skills — those skills associated 
with performing the tasks of particular occupations, 
vocations, or professions. 

• attitudes and values , including 

- attitudes about work and lifelong learning, including 
motivation 

- satisfaction with the educational experience 
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- tolerance for cultural diversity and understanding of the 
values and experiences of other cultures 

- basic values such as honesty and responsibility 

- individual goals and their level of fulfillment 

• involvement and success in subsequent endeavors , including 
• employment 

- additional education (in boch formal and informal settings) 

- professional activity and contributions 

- contributions to community and society. 

These categories with their associated subdivisions describe a full 
array of potential student outcomes, and remain valid whether the unit 
of analysis is a single student or an identified group of students 
(for example, by personal attributes such as age or ethnicity). 
Outcomes of this kind can also be aggregated as performance 
characteristics for institutions or programs. 

2. State-'Level Outcomes Dimensions 

For the state as a whole , an entirely different set of desirable 
conditions are of interest, and are potentially affected by the system 
of higher education. These outcomes often are at the foundation of 
statements of intended impact for the statewide system of higher 
education. They include: 

• access to and participation in the state systeii of higher 
education on the part of individuals of differ(/nt 
characteristics and backgrounds, including 

- socioeconomic status 

- ethnicity 

- gender 

- place of residence (urban, rural, substate region) 

- ability level 

• an educated citizenry 

- literacy levels 

- general level of education 

• a pool of trained manpower to meet the requirements of 
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- the private sector as a whole and by region 

- the public sector as a whole and by region 

- certain identified strategic industries 

• the state's economy as a whole 

- size and level of activity 

- growth 

- stability 

- regional balance 

- employment by occupation and region 

• "quality of life " in the state, including 

- cultural and recreational opportunities available 

- quality of the physical environment 

- quality of the elementary/ secondary educational system 

- quality and coverage of public services 

- access to a variety of consumer goods and services 

3» Research and Service Outcomes 

In addition to positive changes in the characteristics of students and 
groups of students and in the state generally, the higher education 
system prcducPL. important additional outcomes. Traditionally^ these 
are treater^ under the headings of research and service. In thi& case, 
the entities affected are the disciplines and special sub-populations 
within the broader rubric of the state. 

• research/ scholarship outcomes, including 

- new discoveries/ the creation of new knowledge/expanding the 
bodies of knowledge in the disciplines 

- syntheses— organizing previously discovered knowledge in new 
ways 

- applications — applying knowledge toward the solution of 
particular problems or the development of new products or 
methods 

- creative works — the creation of new works of literature, art, 
or music 



• service outcomes, including 

- access to college and university resources on the part of 
citizens and groups within the state or within a region 
within the state 

- improved conditions for identified sub populations, for 
ezaunple 

• health of the indigent 

• service to elderly or confined populations 

4» Dimensions for Assessing the Condition of the State System of 
Postsecondary Education 

Questions about the condition of the state system as a whole are 
generally answered through identifying the characteristics of 
institutions and programs that comprise it. In addition, however, the 
condition of the state system depends upon: 

• diversity of institutions — mix of institutions of different 
types 

• geographic location of institutions 

• mix and location of programs 

• presence and effectiveness of articulation arrangements (the 
relationships among component institutions and programs) 

• mix of alternative delivery mechanisms. 

3* Dimensions for Assessing the Condition of Institutions 

The outcomes produced by institutions are primarily visible in the 
improved skills and capacities of students and in the research and 
service outcomes produced. However, in assessing institutional 
effectiveness, it is also important to assess what the institution 
is— the appropriateness, adequacy, and quality of its assets, both 
tangible and intangible. Dimensions to be considered in this context 
are: 

• clarity of the institution's mission 

• consistency of program offerings with the institution's mission 

• characteristics of the student body , and the match between 
their needs and the institution's structure, policies, and 
procedures 

• cohesiveness and integrity of the curriculum 

• adequacy, appropriateness, and quality of resources , including 
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- faculty and staff 



- facilities 

- equipment 

- library and other collections 

- finances 

• presence and functioning of key institutional processes , 
including 

- plemning 

- resource allocation 

- admissions/recruiting 

- staffing 

- program review/curriculum revision 

- student assessment 

• intangible assets i such as 

- internal organizational climate — morale, openness of 
communication, shared priorities 

- image with critical external constituents (funders, the 
public, the legislature, etc.) — goodwill 

These are dimensions that have historically been the focus of the 
institutional accreditation process ♦ Thsy remain important 
considerations. However, most accreditation agencies are balancing 
the '^asset" perspective represented by these dimensions with an 
increased interest in the actual outcomes produced. 

6. Dimensions for Assessing the Condition of Programs 

The primary outcomes produced by programs are embodied in students in 

their knowledge, skills, attitudes and values, and success in 
subsequent endeavors. These outcomes provide a basis for determining 
the quality of what the program does . In assessing the condition of 
the program itself or its infrastructure, it is appropriate to focus 
on the following areas: 

• clarity of program purpose 

• centrality of programs to the mission of the institution within 
which it is housed 

• coherence and integrity of the curriculum 



12 

21 



• attributes of student majors and the match between their needs 
and program characteristics 

• adequacy and appropriateness of resources, including 

- faculty ard staff 

- equipment 

- facilities 

- library, museum, and other collections 

- finances 

Perspective: Effectiveness Is in the Eye of the Beholder 

Earlier in this paper, we defined effectiveness as achieving desired 
outcomes or conditions. In the previous section, moreover, an array of 
potentially desirable outcomes was presented in summary form. Even in 
summary form, however, the list is very long. If the detail needed to 
make it operational in all its facets were provided, the list would be so 
long as to render it unmanageable. The good news in this situation is 
that all potential outcomes or conditions are not of equal value. Forced 
to choose among them, most individuals could assign rough priorities to 
these items with relatively little difficulty. The bad news is that 
particular priority lists developed by different individuals (or by 
individuals with different roles in the system) will probably be quite 
aifferent. This reflects the pluralistic nature of higher education. 
Students come to higher education to obtain increased personal capacities 
and the economic success that has historically been correlated with 
acquiring those capacities. Industry desires access to the human 
resources of higher education (faculty and researchers), and to the 
products of applied research endeavors^ College administrators and 
faculty typically are concerned with the adequacy and quality of the 
resources available to them. State legislators espouse access, 
efficiency, and contributions to economic development. 

In some cases, all parties at interest will place a high priority on a 
particular outcome — the development of higher order skills in all 
students, for example. In other cases, different constituents will value 
different outcomes but these different priorities complement one another. 
.The students' interest in a well-paying job after graduation, for example, 
is not in conflict with the employer's interest in hiring well-qualified 
new employees. In still other cases, a particular constituent may value a 
particular outcome highly while no other constituent registers interest. 
Finally, there are instances where two or more critical constituents are 
in direct conflict over a basic value. Discipline faculty, for example, 
may maintain that knowledge of a particular discipline is the best 
foundation for success in later professional study, while students and 
ultimate employers may feel otherwise. Given this complexity, those being 
held accountable must make choices. Each must establish a vision of what 
"effectiveness" means in their own context; in short, they have little 
choice but to accept the mantle of leadership. 
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By assuming a leadership role, however, institutional and state agency 
policymakers are faced with an additional problem. They must not only 
identify those dimensions of performance considered ol highest priority 
but mxist also establish standards of performance. The challenge is to 
describe a performance profile for that portion of the postsecondary 
education system for vhich they are being held accountable. 

Sifliplistically stated, educational leadership is revealed in an ability to 
sper-lfy those dimensions of performance having the highest priority (to 
label the horizontal lines on figure 3) and to articulate the associated 
standards or expectatiora (to place the x's on these lines). 

Since being effective means achieving desired ends, it is necessary to 
move beyond stating expectation to an assessment of the extent to which 
those expectations have been met. In this regard two points must be 
noted. First, issues of assessment can take on a life of their own. In 
the technical world, debates become endless about whether or not various 
assessment techniques reliably measiire the extent to which a particular 
outcome has been achieved. While these debates may lead to improved 
assessment techniques, it is important not to allow such considerations to 
dominate discxission. A useful perspective is to adopt the lawyer's 
approach to the topic and look for a preponderance of evidence rather than 
the precision measurements that might reflect an engineer's point of view. 
Using this standard, there are ways to gather assessment information 
relevant to nearly all performance dimensions that might be selected. A 
general discussion of approaches to assessment of various kinds of 
outcomes is inclxided in Appendix A. 

Second, producing outcomes is not the same as being effective. If ^ the 
institution or program persists in producing outcomes that no constituent 
values, that institution or program is not being effective. It is the 
match between actual outcomes and desirable outcomes that determines 
ef f oc tiveness . 

D. The Communications Component 

Being accountable means not only performing up to expectations, but also 
implies commiinicating evidence of performance to those external groups 
that have a legitimate right to know. Briefly stated, the requirement for 
communication involves three steps: 

1 • Identifying the information needs of various audiences. 

2. Acquiring the performance information responsive to those needs. 

3» Presenting that information to various audiences in an 
appropriate, understandable msinner. 

As noted earlier, different constituent groups have different priorities 
for and different perspectives about higher education. Their interests 
are not served by developing a uniform "fact book" and making it available 
to one and all. Rather they are served (and accountability is served as 
well) when the information most critical to them is presented directly in 
its most usable form. For example, potential students, as educational 
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Figure 3 
A Profile of Effectiveness 
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consiimers, need information on which to base decisions about college 
participation and choice. They are served best by information that 
explicitly describes the past outcomes of attendance at a particular 
institution for students irith similar backgrounds and abilities. What, 
for example, can they expect as a probability of successful completion? 
What happens to the graduates in terms of employment or further education? 
Quite different information must be pzresented to legislators; here the 
relevant concern is not only with successful job placement, but with the 
proportion of those job placements occurring within the state, within a 
particular geographic region, or in a particular key occupation or 
industry. 

Because communication is an integral part of accountability, institutions 
and state agencies must explicitly recognize and plan for the targeted 
communications tailored to different audiences. Foremost among these 
audiences are state government, current and potential students and their 
parents, fjid the general public. Additional audiences, arising on a 
periodic basis, are accreditation agencies, the federal government, 
employers in various industries, and particular geographic communities. 

At the same time, system-level accountability requires a different kind of 
communication, this time directed at state policymakers at the highest 
level. Here the object of communication goes beyond demonstrating the 
strengths and weaknesses of the state's higher education system, to 
include an assessment of the relative cost-e/fectiveness of investing in 
higher education compared to alternative investments that the state might 
make. In meeting the manpower requirements of strategic developing 
industries, for. example, states may choose policy approaches that stress 
recruitment of out-of-state talent. In promoting "quality of life" they 
may choose to invest in major cultural institutions, or in improved 
recreational resources. In each instance, it is incumbent upon those 
responsible for higher education to make the case that investment in 
higher education will yield a greater long-term return on such wider state 
objectives than would investment in other alternatives. 

If, as we have argued, communication is a part of accoiontability, 
assessment of how well such communication is accomplished should also be 
undertaken. To date, assessing the communications function has received 
little attention compared to assessing performance, per se . In the design 
of a total accountability system, however, developing an effective 
communication component, and the periodic assessment of how well that 
component functions, must also receive significant consideration. 

E» Institutional and Agency Accouotabilityt A Summary of Roles and 
Responsibilities 

Although the explicit boundaries of responsibility between Colorado 
institutions and CCHE, as in many states, are negotiated within 
constraints of established statute and past practice, their respective 
domains contain areas of sole accoimtability and areas where 
accountability overlaps. A critical issue is the clear determiriiation of 
areas of both independence and interdependence. 



Turning to some of the concrete contents of each dooain, figure 4 contains 
an indication of priority areas of responsibility for institutions and 
state agencies. This table is provided as an illustration, not a 
recommendation for the state of Colorado. 

Cer-^ainly, we do not intend this clapsification to be prescriptive. 
Rather it represents our initial identification of those areas most 
central to the accountability concerns of each party, based on current 
national practice. At the most basic level, this classification 
highlights the fact that a state agency's primary attention should be 
devoted to shaping the state's higher education system as a whole, to 
identifying those areas in which the state has an identifiable 
programmatic (manpower production or research/ service) priority, to 
ensuring the adequacy of resources provided, and to ensuring overall 
efficiency as well as effectiveness. 

Institutional administrators, in turn, are held responsible for ensuring 
that all students acquire general knowledge and skills, that programs 
produce students with the requisite special knowledge and skills, that 
adequate and appropriate resources are acquired, that a student body of 
desired characteristics is recruited and retained, that programs 
consistent with institutional mission are offered, and that the 
institution operates efficiently. Clearly there are points of overlap 
between the two parties, and considerable attention must be given to 
coordination and developing common understanding. Just as importantly, 
however, there are significant areas of largely independent 
responsibility. Care must be taken that these areas are not neglected in 
an environment in which much energy is given to the areas of joint 
interest and accountability. 
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Figure 4 
Typical Locus of Responsibility 
for Different Areas of Higher Education Performance 



Domain 



Substance 



Student Outcomes — All Students 

General Knowledge (Modes of 

Inquiry) 
Basic and Higher Order Skills 
General Values 
Relationships 

- To Community 

- To Economy /Employment 
Student Goal Achievement 

Retention 



Student Outcomes — Majors in Programs 

Special (Discipline Specific) Skills 
Special (Vocational/Professional) 
Skills 

Values and Ethics Pertinent 

to Profession 
Subsequent Success in 

- Employment 

- Further Education 

- Profession 

Student Outcomes — Individual Students 

Minimal Levels of 

- General Knowledge 

- Basic and Higher Order Skills 

Specific Knowledge 
Specific Skills 



State Outcomes — 

Access/Participation 

Educated Citizenry 

Pool of Trained Manpower 

Employment Levels and Distribution 

Economy 

Quality of Life 



Institution 



X 
X 
X 

X 
X 
X 
X 



State Agency 



X 
X 



X 
X 



x-all programs 
x-all programs 



x-all relevant progs, 
x-all, as appropriate 



X particularly 

for those 
X programs defined 

as being a state 
X priority 
X " 



x-all students to 
be granted a 
degree 

x-all students 
x-granted a 

disciplinary degree 



x-only those 

students certifiei 
licensed by the 
state 

X- ti 
X- " 



X 
X 
X 
X 

X 
X 



27 

18 



ERIC 



Figure 4 
(Continued) 



Domain 



Substance 
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Figure 4 
(Continued) 



Domain 



Substance 
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III. THE RELATIONSHIP OF ACCOUNTABILITY TO OTHER STATE PROCESSES 



A. Introduction 

Processes for ensuring accountability in higher education do not exist in 
a vacuum. While the substance of accountability revolves around assessing 
and determining responsibility for ends or conditions, many other 
relationships exist between state higher education agencies and 
institutions. These relationships are primarily means-oriented; they 
represent particular policy levers available to state agencies by means of 
which they seek to influence the higher education system. Among these 
state-level mechanisms are: 

1 • Master planning 

2. Establishing institutional mission, role, and scope 

3» Program review 

A* Financing/state-level resource allocation/budgeting 

5. Governance — establishing roles and responsibilities of the various 
parties at interest. 

Each of these processes is part of a particular operational task that 
needs to be performed. Together, however, they provide the state higher 
education agency with means to influence the performance of the system as 
a whole. 

The purpose of this section is to explore some explicit links between 
these established mechanisms and accountability. Our reason for 
documenting these links is a compelling need to ensure that overall policy 
formulation be undertaken consistently. Typically, we find that policy in 
each of these areas is developed independently. Indeed, in many cases 
such statewide processes are developed in direct response to a particular 
external pressure or mandate rather than in response to any articulated 

grand design" for policy. Furthermore, rarely are such processes 
developed with a view toward how they will fit together administratively. 
Often, this is because their development reflects current organizational 
arrangements in both institutions and agencies — domains in which academic 
people devise program review mechanisms, finance people devise resource 
allocation schemes, and so forth. 

Too often, the primary result of such unrelated approaches to policy 
formulation is confusion. Incentives built into the budget may or may not 
reflect the priorities established in the master plan. Criteria employed 
in program review may or may not include important dimensions stressed by 
the accountability process. In the face of contradiction, the primary 
response tends to be to rely on bureaucratic procedure rather than on a 
shared understanding to keep the system functioning. 

Certainly the following sections are not intended as cure-alls for this 
common malady. Rather, they are intended to identify points of tangency 
between accountability and other salient and established state agency 



22 



EKLC 



• 30 



processes and functions. Our ultimate aim in this discussion is to 
illustrate how wrong it is to view accountability as simply another 
process. Often, in fact, it will function through many established state 
agency processes to achieve a common end. 

B. Master Planning 

Master planning is the state-level equivalent of strategic planning at the 
institutional level. Here the linkage to accountability is direct, as 
planning is the visible mechanism by which state priorities are 
established and levels of expectation set. At the core of master planning 
is the need to specify: 

• Those outcomes to be produced by the system of education that have 
the highest priority 

• Those characteristics and capacities of the state higher education 
system that have the highest priority . 

In both cases, the categories among which priorities must be set are the 
same as those presented earlier as potential dimensions of performance for 
accountability. To briefly reiterate, the outcomes of potential interest 
are; 

1. Access/ participation in the system of higher education 

2. An educated citizenry (literacy and general levels of education) 
Trained manpower in selected fields 

4. Employment levels and distribution 
3* Economic growth 
6. Quality of life 

7* Research and scholarship outcomes in selected areas 
8. Provisions of specified services 

Similarly, the characteristics and capacities of the state system at issue 
in any master planning exercise are: 

1 . Institutional diversity - the relationship and role of public 
education vis-a-vis independent and proprietary institutions, and 
the missions of the various institutions 

2. Geographic location 

3* Mix and location of programs 

4» Interrelationships among institutions and programs 
3* Mix of desirable delivery mechanisms 
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6. Institutional and program quality - how nationally competitive 



?• Adequacy and appropriateness of resources 
8. Efficiency 

The assumption here is that master planning involves a series of 
state-level choices about what the system should do and should be. We 
xrecognize that this conception is at variance with much master planning as 
historically practiced. Traditionally, the focus of such exercises is 
confined to predicting enrollment levels in various institutions and to 
calculating the resource requirements associated with each enrollment 
level — a largely reactive exercise. 

To reiterate, Master Planning is a necessary precursor to any state-level 
accountability process. It establishes both the framework and the 
criteria within which an accountability process must function. In its 
absence descriptive accountability information can be generated, but zhe 
core question of the system's effectiveness cannot be addressed. 

C. Establishing Institutional Mission, Role, and Scop^ 

This is perhaps the most contentious and difficult task faced by most 
state higher education agencies. Each term is imprecise and any action 
taken to specify mission, role, and scope is more likely to be seen by all 
parties as confining rather than enabling. Nevertheless, it is a task 
frequently assigned to the state higher education agency, and one directly 
related to accountability issues. 

A statement of mission, role, and scope is a bounded statement of 
institutional aspiration. If written appropriately, it provides license 
for an institution to become more than it currently is, while at the same 
time, it puts some identifiable constraints on the dream. Caruthers and 
Lott state that mission, role, and scope statements describe the "static 
identity" of the institution — specifying basic institutional philosophy, 
clientele served, and services provided. Here again, however, major 
questions involve determining: 

1 . What the institution will do - the kinds of outcomes it seeks to 
produce and the student goals best served by the institution, 
including 

• knowledge 

• skills 

m attitudes/ values 

• subsequent relationships 

2. What particular clientele(s) the institution is to serve, stated 
in terms of 

• geographic origin 

• ability/academic achievement 

• age 
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• full- time/part* tine status 

• employment in certain professions (e.g., teaching or health 
care) 

• socioeconomic status 

3* What the institution is to be, stated in terms of 

• breadth and depth of instructional offerings 

• research or scholarship priorities 

• array of services to be provided 

• a collection of assets of particular (comparative/absolute) 
quality 

• a particular kind of educational environment 

( caring/ competitive/ theoretical/ practical/ intellectual/etc . ) 

• a reflection of a particular set of values (professional 
preparation, liberal arts, etc.) 

While these dimensions describe an institution's mission, the notion of 
"role" as separate from "mission" is seldom adequately addressed. From an 
accountability perspective, we suggest that "role" be used to specify any 
particular responsibilities explicitly assigned to the institution in 
achieving an identified state priority. Examples might include general 
literacy, or service to particular industries, professions, or other 
constituencies. "Scope," in contrast, we suggest be used to specify a 
constraint — for example, the specification that a particular institution 
undertake only undergraduate instruction. A statement of institution 
mission, role, and scope, if stated well, will therefore indicate what an 
institution is not as well as what it is. 

Statements of mission are thus the mirror image of the performance 
dimension constituting the heart of accountability. If carefully written, 
statements of institutional mission direct the assessment of effectiveness 
at both institutional and state levels. In the absence of a clear 
statement of mission, selecting particular measures of effectiveness 
itself creates a de facto statement of mission. By selecting particular 
measures and eschewing others, hidden priorities are revealed. Without 
the rationale provided by a mission statement, however, these priorities 
may or may not be appropriate. In the absence of either a mission 
statement or an assessment plan, of course, institutional behavior 
operationally defines the "mission." 

Two implications for accountability emerge from this discussion. First, 
accountability and its associated assessment procedures, must reflect the 
full range of an institution's mission. For a major research university, 
for example, accountability should be cast in terms of the research and 
service components of its mission as well as covering its instructional 
components. Second, in the absence of an agreed-upon mission statement, 
accountability or assessment processes should not covertly encourage or 
reveal a scope that is inconsistent with that assigned to the institution. 

D. Program Review 

Program review, as a statewide exercise, has two distinct functions. The 
first is to appropriately adjust the complete inventory of programs 




offered by the state system. In some instances this means adding a new 
program; in other instances it means discontiniiing a program currently on 
the books. The second is to evaluate ongoing programs with an eye toward 
individual program improvement. The appropriate role of the state agency 
in the first of these tasks is very different from its role in the second. 

In decisions affecting the inventory of programs — program approval and 
discontinuance — state agencies typically have a direct role. Indeed, it 
is qxiite common for a state agency to establish criteria for approving all 
new programs, and to have final authority for approving institutional 
requests to initiate new programs. Many state agencies, moreover, have 
the authority to recommend (or order) institutions to discontinue 
programs, xasually after establishing public criteria for such action. 
Given its responsibility for statewide issues, we suggest that CCHE derive 
criteria for program approval or discontinuance directly from the 
statewide performance dimensions identified previously. Those most 
important to recall ares 

1. State need — contribution to state system effectiveness. In most 
cases this criteria applies to professional programs through the 
preparation of needed manpower; there are numerous instances, 
however, where programs are added to increase participation rates 
for partictdar groups of students. 

2» Furtherance of assigned institutional mission — ^'contributions to 
the mission, role, and scope established for each institution 
throughout the system. Similarly, programs may be discontinued 
because they are determined to be outside the institution' s 
approved mission or scope. Offerings of two-year programs in 
predominantly four-year institutions may be of this nature » 

3. Impact on system-level efficiency— what are the marginal costs of 
having or not having the program? Here, the criteria are often 
expressed in terms of program productivity and demand (number of 
majors and numbers of degrees granted per year) or duplication of 
programs. More important, however, is an added requirement to 
indicate that operating costs would in fact be reduced if the 
program were eliminated — and in many cases would not significantly 
increase if the program were approved. 

Turning to the second function of program review— that of improving 
existing programs— the state role is significantly different. Here the 
role is not judgmental; rather it includes the following tasks: 

1 . To encourage or ensure the presence and effective fionctioning of 
an appropriate local review process at each institution 

2. To encourage or ensure inclusion of those program perfomance 
dimensions of interest to the state in assessment guidelines 
governing the local review process. 

Regardless of the purpose of program review, the state higher education 
agency is faced with a need, .to identify criteria for action. In both 
cases auch criteria are already identifiable as dimensions of 
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accountability. When the function is program approval/discontinuance, 
these criteria must be made explicit; failure to do so inevitably leaves 
the agency open to changes of arbitrary and capricious action. Where the 
function is program improvement, moreover, the dimensions of performance 
of interest to the state primarily include the contributions of the 
program toward achievement of identifiable statewide priorities. Failure 
to make such contributions can be reviewed through the process and action 
taken to redress the problem. Here the state role is to make its 
priorities known, without interfering with local action directed toward 
progrsun improvement. 

E. Budge ting/ Re source Allocation 

The budget represents a principal avenue by which policy initiatives are 
implemented by state government. Because resource allocation mechanisms 
contain incentives for positive action on the part of institutions, the 
budget is a particularly attractive vehicle for shaping behavior. Thus, 
it is important to recr^gnize linkages between the budget and achieving 
priority accountability objectives. 

All procedures and mechanisms utilized to allocate resources inherently 
contain a set of incentives (and disincentives) for particular types of 
institutional behavior. In the main, however, these incentives remain 
implicit. In the drive to develop mechanisms that allow for the equitable 
allocation of resources to institutions, the incentives buried in the 
mechanism are seldom acknowledged. Instances are currently infrequent in 
which resource allocation procedures are designed explicitly to provide 
incentives for achieving selected, desirable ends. Most state-level 
resource allocation mechanisms are based on student FTE enrollment—either 
directly through formula or indirectly through incremental adjustments to 
a base budget figure. Here the incentive to the institution is clearly to 
enroll more students. To the extent that the allocation mechanism also 
reflects program cost differentials, there are additional incentives to 
expand high cost programs to the limits of student demand. 

If the primary state goal is to expand access, this is a highly effective 
mechanism. If the intent is different, however, it is important to 
recognize sevaral secondary incentives that are corrolaries to any 
FTE-driven formula. On the plus side, formulas provide incentives to 
increase the number of students retained. On the down side, such 
procedures create incentives to lower admission standards, to develop, 
without regard xo mission, a wider array of programs in an attempt to 
attract a larger clientele, and in some cases to lower academic standards 
in an effort to avoid driving off any students that have enrolled. Other 
frequently used approaches to state-level resource allocation seek to 
ensure institutional capacities in certain areas deemed important, for 
example the capacities to deliver cooperative extension services, health 
care to the indigent, etc. These ends are typically achieved through 
special-purpose allocations of funds made to specific institutions. 

The bulk of state funding provided to institutions of higher education 
will continue to be allocated through procedures that implicitly or 
explicitly establish increased student access as a state priority. 
Nevertheless, there are numerous opportunities at the margin to provide 
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incentive funding in support of additional high priority state objectives. 
• Among the possibilities are: 

1 . Use of capitation grants to create incentives for producing 
manpower in selected critical need areas. 

2. Use of similar mechanisms to reward institutions for helping 
minority students successfully complete a program of study. 

3. Use of competitive grant programs to develop centers of excellence 
in areas deemed to be important to fill the research and service 
needs of the state. 

4. Allocation of a pool of applied research enhancement funds to 
institutions on the basis of the amount of support for such 
activities acquired from the private sector. 

5« Creation of a pool of resources to match funds raised by 

institutions in support of particular services or capacities 
deemed .Important by the state. 

Points important to accountability in this brief discussion are two. 
First, all methods of resource allocation inherently contain a series of 
incentives and disincentives. It is important to catalogue those embedded 
in the resource allocation mechanisms currently being used and to 
ascertain the extent to which they are consistent with the achievement of 
priority state goals. Second, the incentive power of the budget is seldom 
effectively utilized. Once state priorities are established, creative 
uses of the resource allocation process to create incentives for the 
achievement of those desired ends can be a powerful policy lever for 
accountability. 

F. Governance 

Accountability, as we have defined and discussed it, also has major 
implications for governance relationships. To be sure, the structural 
aspects of each such relationship will be unique to each state and will 
reflect its history, traditions, and political culttire. This discussion 
cannot, and should not, deal with this subject. Several general 
implications for governance relationships, however, cannot be ignored. 
They include the following: 

1 . The state higher education agency ijhould assume responsibility for 
articulating statewide priorities for education, ensuring 
attention to these priorities by the governing boards and 

i: Institutions, and monitoring progress toward achieving them. 

2. Each institution should identify those dimensions of performance 
that best reflect its mission and should establish a mechanism to 
assess performance along those dimensions. The state role is to 
ensure that this selection reflects state priorities where 
appropriate, and to monitor the assessment process—not to do it, 
but to ensure that it is done. 
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3. The state agency should be responsible for communicating statewide 
accountability information to relevant statewide external 
constituents, primarily the public and other branches of state 
government. The institutions and governing boards should be 
responsible for communicating their appropriate accountability 
infonnation to their own constituents — potential students and 
their parents, the public, state government. In this second 
regard the state role again is to ensure doing, not to do. 

4» The state higher education agency should be responsible for 

ensuring that the various processes it administers are internally 
consistent, serve to reinforce achievement of identified 
state-level priorities, and contribute to a state system thar 
contains institutions of complementary function and 
responsibility. 

While these implications are stated in general terms, they serve to 
delineate some particular roles and responsibilities of major actors. 
Perhaps most usefully, they attempt to further articulate the special 
roles and responsibilities of a state agency. Although appropriate 
accountability responsibilities of public institutions have been stated in 
other places (e.g., NGA and ECS reports), the explicit state agency role 
has largely been defined to be one of ensuring institutional compliance. 
The above identifies a second major role — being accountable for the 
performance of the state system of higher education. 



IV. APPLICATION OF CONCEPTS TO THE STATE OF COLORADO 

In the first two sections, our intent was to develop the notion of 
accountability conceptually, without dealing explicitly with its application to 
conditions and needs within the state of Colorado. This task accomplished, we 
can now turn more specifically toward defining explicit requirements for 
accountability processes and related state agency functions in the state of 
Colorado. In this section, we will exeunine a variety of Colorado statutes and 
procedures in the light of previoxisly developed concepts. In the section that 
follows, we will build on these observations to prepare a series of 
recommendations for action. 

A. Higher Education Accountability 

Article 13 of HB1187 describes the intent of the legislature with regard 
to establishing a higher education accountability program for the state of 
Colorado. In this statute, the legislature dealt explicitly (if briefly) 
with the substance of accountability, and with the communication of 
accountability information. With regard to the former, HB1187 states 
that: 

"Institutions of higher education (will) be held accountable for 
demonstrable improvements in student knowledge, capacities, and skills 
between entrance and graduation." 



In addition to noting "knowledge, intellectual capacities and skills," the 
statute indicates that 

"expected student outcomes . • . may include other dimensions of 
student growth, such as self-confidence, persistence, leadership, 
empathy, social responsibility, understanding of cultural and 
intellecttial differences, employability, and transferability." 



Comparing this statement of desired student outcomes with the 
classification presented in section II highlights the fact that 
substantial additional specification of intended outcomes must be 
developed before assessment issues per se can be addressed. Questions of 
"what knowledge, which intellectual capacities and skills are of paramount 
importance?" must be addressed in detail. In the absence of specific 
language to the contrary, we conclude that institutions have considerable 
latitude in identifying those particular dimensions of student outcomes 
most important to them, consistent with 1) their institutional missions 
and 2) the policy directions of CCHE. Despite substantial freedom in 
selecting those outcomes dimensions that best reflect their missions, 
however, we note that by use of the word "knowledge" in Article 13, the 
General Assembly explicitly mandates assessment of cognitive outcomes. By 
extension, accountability approaches that focus solely on student 
placement or other aspects of behavior will be found deficient. 

Furthermore, HBn87 deals explicitly only with institution-level outcomes; 
the concept of system-level outcomes is not recognized in statute. In the 
extant Master Plan for Colorado Postsecondary Education, however, five 
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primary goals for the postsecondary system are proposed. This set of 
goals is also incorporated into section 23-1-108 of HB1187. Somewhat 
paraphrased, the goals include: 

1 . Quality , where quality includes 

- Faculty with high academic or professional achievement and 
demonstrated competence; 

- Students with the ability, preparation, interest, and motivation 
to benefit from postsecondary education 

- Curricula with rigor, including for degree programs, a sound 
ba*^ in the liberal arts and sciences; 

- Learning support systems, including libraries, laboratories, 
learning technologies, and facilities; 

- Evaluation to assure quality; 

- Participation by faculty in research appropriate to the role and 
mission of the institution; 

- A financing base sufficient to assure minimum standards for the 
preceding elements 

2. Access, as manifested in the opportunity of any individual in 
Colorado, with appropriate ability, preparation, interest, and 
motivation to attend an undergraduate program somewhere in the 
state. Limits to access to be overcome are specifically noted as: 

- Financial 

- Geographical 

- Remedial 

3* Diversity of educational opportunity as manifested in an array of 
institutions of different types providing diversity along the 
following dimensions: 

- Academic and occupational programs and institutions with 
specialized programs; 

- methods of learning; 

- academic competitiveness; 

- size; 

- residential ar.d commuter campuses; 

- public, private, and proprietary institutions. 

4. Efficiency— accomplishing above goals within the limited resources 
available to the system of postsecondary education 

5. Accountability for actions ^d policies to the elected 
representatives of Colorado citizens, explicitly including 
procedures for: 

- Program review 

- Financial review 

- Personnel policy review 

These goal statements strongly reflect an orientation toward "system 
assets" as defined in previous sections. Of the list of "state-level 




outcomes" defined earlier, only "access" is treated in the current master 
plan; outcomes dealing with such topics as trained manpower, the economy, 
and quality of life are currently omitted. Furthermore, questions 
concerning which, if any, student outcomes might be of priority interest 
to the ata-ce remain unanswered by the Master Plan. 

Article 13 of HB1187 also deals with some of the procedxiral issues 
associated with conducting an accountability program. Items of particular 
note include: 

1. Evidence of demonstrable improvements in knowledge, capacities, 
and skills of stixdents between entry and exit is required. In 
short, where appropriate, there is an expectation that educational 
"value added" ^e assessed. 

2. No particular assessment techniques are mandated; the only 
requirements are that assessment techniques be appropriate to the 
outcomes being assessed and that they be employed on an ongoing 
basis. 

3. The information gleaned through this process be utilized to 
improve those programs to make outcomes more consistent witJh 
expectations. 

Again the focus of accountability is on institutions; similar expectations 
with regard to CCHE and its accountability role vis-a-vis the higher 
education system are not expressed. 

With regard to the communications aspect of accountability, HB1187 
mandates that "these demonstrable improvements be publicly announced and 
available." The law further states that "the results of the institutional 
assessment shall be communicated to the public, its students, and 
potential students •" Finally, CCHE is directed to "report annually to the 
Governor and the General Assembly on the development and implementation of 
this article." In this case, reporting requirements are established for 
both institutions and the state higher education agency. With regard to 
the former, the requirement emphasizes substantive aspects of 
institution-level accountability; with regard to the latter the emphasis 
is on procedural components. The requirement for a periodic report from 
CCHE to the public and ti;<? General Assembly on progress toward achieving 
substantive goals for the system of postsecondary education in the state 
is not explicitly addressed. 

B. Master Planning 

Section 23-1-108 of HB1187 deals with the duties and powers of CCHE with 
regard to systemwide planning. It states that "the commission shall 
develop and submit to the Governor and the General Assembly a master plan 
for Colorp.do postsecondary education." The legislation further states 
that, as part of the master planning process, the Commission has the 
authority to: 

1. Establish a policy-based and continuing systemwide planning, 
programming and coordination process to effect the best use of 
available resources; 
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2.* Establish such academic and vocational education planning as may 
be necessary to accomplish and sustain systemwide goals of high 
quality, access, diversity, efficiency, and accountability; 

3« Determine the role and mission of each state- supported institution 
of higher education with statutory guidelines; 

4« Establish enrollment policies; 

5. Establish state policies that differentiate admission and program 
standards and are consistent with institutional role and mission; 

6. Adopt statewide affirmative action policies for the Commission, 
governing boards, and state- supported institutions; 

7. Develop crJt^/la for determining if an institution should be 
consoiidatv:: ot- closed. 

This mandate overlaps several other assigned CCHE responsibilities, 
specifically role and mission definition, program review, and resource 
allocation. In identifying this range of responsibilities, however, the 
statute reaffirms the five systemwide goals previously established in the 
Master Plan—quality, access, diversity, efficiency, and accountability. 

It is also important to understand that a process to review and revise the 
Master Plan is currently underway. As part of this process. Commission 
staff have conducted numerous roundtable discussions at sites throughout 
the state to determine ways in which higher '^lucation coiild assist in 
meeting regional ('economic and other) goals. These and other activities 
will probably result in revision or extension of existing goal statements 
included in the Master Plan. At the same time, goals dealing with either 
student outcomes (important skills etc.) or priority state outcomes 
(economic change, manpower development) might also be added. Selection of 
these priority areas is perhaps the most important task facing the 
Commission in the Immediate future. Their specification represents the 
linch-pin that cohesively integrates niimerous other Commission processes, 
among them role and mission definition, resource allocation, and program 
review. Niimerous other policy initiatives are incorporated under the 
umbrella of master planning. Reexamination or reaffirmation of a 
statement of syatemwide goals is central to many of them. As this process 
takes place, an important question involves the extent to which any 
revised statement can explicitly identify what the system is to do as well 
as what it is to be. 

C. Institutional Mission, Role, and Scope 

Among the provisions of HB1187 are statutory statements of institutional 
mission for each of the four-year institutions of higher education and for 
community colleges generally. These mission statements distingxiish 
institutions primarily on the basis of the breadth and depth of programs 
they are expected to offer and of the admission standards applicable to 
the institutions. Having set statutory bounds in this manner, the 
legislature directed CCHE to further define the role and mission of each 



institution and to establish for each, appropriate geographic and 
programmatic service areas* * 



Some institutions such as the University of Colorado at Boulder, Colorado 
State University, School of Mines, and University of Northern Colorado are 
explicitly identified as statewide institutions. Others are presumed to 
have a regional service area. Even those institutions having a statewide 
service area are expected to serve limited programmatic roles. For 
example, UNO is responsible for providing "graduate level programs needed 
by professional educators and education administrators." Similarly CU and 
CSU are charged with the responsibility to provide thoee grad» ate-level 
programs designated by the Commission as part of their respeci&ive 
statewide responsibilities. 

Moving beyond statute, the Commission is faced with the task of 
identifying additional bases for institutional differentiation. In part, 
this will be achieved by further refining the statements of particular 
clienteles to be served by each institution. In some cases, clientele can 
be described in terms of residency within a particular service area. In 
others, clientele can be defined on the basis of profession (teachers, 
health care professionals, etc.). Mission differentiation can also be 
further accomplished by assigning to selected institutions leadership in 
the achievement of particular state^level goals. In any case, linkages 
between state higher education objectives and the nature of the higher 
education system as manifested in the array of assigned institutional 
missions must be recognizable and explicit. Important in this regard are 
citizen comments received during roundtable discussions in support of 
Master Plan revision. These comments indicated strong interest in 1) 
further adapting smaller institutions to serve particular needs of their 
regional service areas and 2) ensuring certain knowledge and skill 
outcomes on the part of all students educated in these institutions. 

D. Program Review 

Various provisions of HB1187 state that: 

1. "The Commission shall review and approve, consistent with 
institutional role and mission and statewide educational needs, 
the proposal for any new program." 

2. "The Commission shall establish . . . policies and criteria for 
the discontinuance of academic or vocational programs." 

3* "Each governing board . . . shall submit to the Commission a plan 
describing the procedures and schedule for periodic program review 
and evaluation of each academic program at each institution 
consistent with the role and mission of each institution." 

By these statutory provisions, the Commission is assigned a direct role in 
establishing criteria for new programs and for programs to be 
discontinued. Conceptually, the kinds of criteria appropriate to such 
processes were identified in section III above. To repeat, they include: 
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1 • state need - contribution to state system effectiveness 

2. Furtherance of assigned institutional mission 

3- Impact on system-level efficiency. 

Given current definitional deficiencies with regard to dimensions of 
state-level effectiveness and statements of institutional mission, CCHE 
cannot have a fully developed set of criteria for program approval or 
discontinuance according to these criteria. By default, significant 
short-run reliance will be placed upon the remaining 

criterion— efficiency. Nevertheless, some guidance can be obtained from 
extant statements of state priorities and institutional missions. For 
example, the extent to which new programs might contribute to improving 
access for underserved student populations is an identifiable review 
criterion derived from existing statements of state priorities. 
Similarly, specific prohibitions contained in statutory statements of 
institutional mission provide some criteria that can and have been 
employed in decisions to discontinue programs. Nevertheless, the amount 
of gxiidance provided in this arena could be substantially increased by 
developing a refined list of state-level priorities and by a set of more 
rigorously defined institutional mission statements. 

The statute also gives CCHE authority to require a plan for systematic 
program review from each institution. The Commission has previously 
promulgated "Policy and General Procedures for State Level Review of 
Existing Programs," that specifies a variety of data elements and 
questions to be addressed in the case of each program reviewed. Included 
are questions of (1) centrality to mission, (2) student interest and 
program potential, -(3) quality of resources, (4) program outcomes defined 
in program-specific ways, and (5) efficiency. These criteria are all 
interpreted in light of institutional, not systemwide, priorities. As 
guides to institutional process, in most- cases, this is as it should be. 
Numerous programs, however, can be expected to contribute directly to the 
accomplishment of state-level goals. To date, CCHE has made no provision 
for incorporating "contribution to state-level priorities" as an explicit 
review criterion. This issue will be moot until the Commission identifies 
statewide priorities to which programs can be logically and directly 
linked. At that time, however, there will be a need tc review these 
criteria. 

Finally, conversations with Commission staff suggest that the current 
outcomes criterion in program review rarely yields substantial data on 
performance. At most, what is reported under this criterion is output or 
productivity information such as ntmbers of degrees granted or units of 
service provided. Occasionally, programs will report placement rates or 
(where easily documented) rates of passing certification or licensing 
examinations. If program review is to be effective in both serving 
accountability and Improving performance, the outcomes criterion must 
receive at least as much attention aa those dealing with student demand, 
centrality to miafsion, and quality of resources. At minimum, pr gram 
review should include some indicators of cognitive or skill outCv^mes as 
well as more substantial and complete information on former student 
success* 
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E. Resource Allocation 



Among the provisions of HB1187 are sections dealing with the role of the 
Commission in appropriating and allocating funds for operating and capital 
purposes. For operating funds, this role is made explicit in section 
23-1-105, that states: 

"The commission shall establish, . . . the distribution formula of 
appropriations by the general assembly to each governing board under 
the following principles: 

a) To reflect the different roles and missions of institutions 

b) To reflect institutional costs which aure fixed and those which 
vary, based upon the character of programs and the numbers of 
students enrolled; 

c) To reflect an emphasis on decentralized financial 
decision-making and stability of funding." 

In addition, special provision is made for a quality incentive grant 
program, supported by an annual appropriation, to "promote, encourage, and 
recognize centers of excellence." 

At the moment, the quality incentive grsuit program is the only part of 
assigned resource allocation responsibilities that directly ties funding 
to the accomplishment of stated objectives. Criteria to be used in 
identifying programs to be rewarded through the Recognition of Excellence 
portion of this program include l) availability of outcomes information 
that demonstrates excellence in achieving a statewide goal, 2) importance 
to the institution's role and mission, and 3) evidence of having met an 
institution/college/ departmental goal. Criteria to be used to select 
projects to be funded through the Promotion and Encouragement of 
Excellence component of the program include l) evidence that the 
activity/project is related to a statewide goal and 2) importance to role 
and mission. These criteria make clear an intended linkage between 
acquisition of resources and demonstrated performance in areas that 
directly serve a state priority and/or that reinforce an institution's 
role and mission. As with program review, any steps taken to sharpen 
statements of state goals and institutional missions will make this 
program easier to administer and will render it a better instrument of 
state policy. 

With the exception of the quality incentive grant program, most resources 
are allocated in a manner that is not linked to either institutional 
effectiveness or to identified contributions to state-level effectiveness 
(except insofar as the incentive to recruit more students serves an access 
goal for the stete). The formula utilized is of a fairly common form, 
designed to allocate funds to governing boards equitably given the 
programmatic and student demands placed on their institutions. As 
Indicated in section III, numerous additional budgetary devices are 
available to more directly tie resources to the achievement of state 
objectives or to reinforcement of institutional mission. As in prior 
sections, however, the instrument can be no sharper than the policy it 
implements. Until and unless statements of state priorities are made more 
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definitive, opportunities to create incentive mechanisms through the 
- budget process will be limited. 

Before leaving operational budgeting, it should be noted that the General 
Assembly has in the past made direct, single line item appropriations to 
governing boards for operation of certain programs of particular interest 
to the state. Mentioned explicitly in statutes are programs at the Health 
Sciences Center and the veterinary medicine program at CSU. In these 
cases, special treatment is probably at least as much a reflection of 
complexity as of special interest. Nevertheless, they set a precedent for 
similar categorical funding approaches to particular programs or 
initiatives deemed to be of importance to the accomplishment of state 
purposes. 

In the area of capital budgeting, the Commission is charged with annually 
developing recommendations concerning "priority of funding of capital 
construction projects for the system of higher education." Section 
23-1-106 further directs the Commission to ensure l) that proposed 
projects are consistent with role and mission master planning of the 
institution, and 2) that facilities master plans conform to approved 
educational master plans. Here again, the institutional mission statement 
is the template against which proposals are ultimately measured. In 
instances where statements of institutional mission are neither clear nor 
commonly understood by all parties, established criteria for project 
funding have the potential for being extremely hard to administer. 

F. Governance 

Overall governance arrangements for higher education in the state of 
Colorado involve considerable decentralization, with significant authority 
residing in the various governing boards. Thus, in implementing the 
educational accountability program envisioned by Article 13 of HB1187, the 
governing b'i.^rds eventiially mxist come to play significant roles. 
Following our development of the accountability implications of this 
article, governing boards are responsible for: 

1 . Ensuring that each of the institutions under their control 

identify those dimensions of institutional effectiveness that are 
of institution-wide concern and reflect that institution's 
mission. 

1. Ensuring that programs and procedures are established to assess 
current levels of institutional performance along each dimension. 

3. Seeing that the results of these assessments are made available to 
the public, students, and the elected representatives of the 
citizens of Colorado. 

4. Taking action to icprove institutional programs and practices 
based on obtained results. 

The role of the Commission in this milieu is threefold. First, a parallel 
set of priority identification and assessment activities at the system 
level necessarily evolve to the state higher education agency. These 
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activities not only provide benchmarks according to which to judge the 
effectiveness of the system but they also provide a necessary context for 
planning and assessment activities at the campus level. Second, there is 
an oversight function that the Commission cannot escape — ensuring that 
governing boards faithfully discharge the four responsibilities described 
above. Finally, the Commission has a special role in seeing to it that 
institutions effectively discharge their particular responsibilities for 
meeting selected state priority objectives. In such instances, the 
Commission's concern goes beyond the need to ensure that institutional 
priorities are appropriately set, assessed, and reported. It also 
includes a legitimate mandate to set certain common standards for 
assessment procedures or for particular outcomes. 
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V. RECOMMENDATIONS FOR FUTURE ACTION 

Previous sections of this white paper described basic concepts of 
accountability, noted many relationships between accountability and other 
state-level processes, and discussed the mandate for establishing a Colorado 
higher education accountability program in the light of these concepts and 
relationships. This discussion provides a foundation for proposing action 
recommendations to the Commission and at the institutional level. The purpose 
of this section is to present these recommendations. All, we trust, are based 
on sound conceptual principles, and reflect the political culture and 
governance structure shaping Colorado higher education. 

As outlined previously, we identify several distinct types of responsibilities 
of state higher education agencies. First, the state agency is directly 
responsible for implementing accountability as it applies to the state- system 
level. Secondly, the state agency has a responsibility for directly ensuring 
the accountability of institutions with regard to selected, statewide 
priorities. Thirdly, the Commission is responsible for establishing guidelin-as 
for local institutional action to fulfill accountability obligations, to 
directly assist institutions in designing and implementing such programs, and 
to appropriately monitor these initiatives as they develop. Recommendations in 
each of these three areas are presented below. Finally, we present some 
additional recommendations for institution-level action. 

A. Recommendations to CCHE with Regard to System-Level Accountability 

Recommendations that apply to the Commission's role at the system level 
are as follows: 

1. That the Commission, through master planning and other assigned 
activities, indicate specific areas of systemwide performance 
considered to be of highest priority. Using the classification 
presented in section II y consideration should be given to 
priorities in the areas of: 

• State-level outcomes 

• Student outcomes that should be a priority for each institution 

• Research and service outcomes 

• Development or maintenance of the higher education 
Infrastructure in the state. 

The essence of this recommendation is that the Commission design 
and implement an accoimtability program focused on assessing 
systemwide priorities. In the process, the Commission will 
acquire direct experience of the kinds of issues being faced by 
institutions and governing boards. At the same time, it will 
develop a coherent statement of state priorities that can provide 
a firm foundation for both statewide planning and for 
communicating important issues to government and the public, and 
to the institutions. 

In identifying system-level performance dimensions, care should be 
taken to state priorities in such a way that they give specific 
direction to subsequent assessment processes. General statements 
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about the importance of manpower trained to meet employer needs, 
for exMple, are less useful than more specific statements about 
the need for individuals with particular skills available to meet 
the needs of explicit occupations or industries. 

That the Commission, working with the governing boards, refine 
existing mission, role, and scope statements for all institutions. 

Statements of institutional role and mission are at the hub of 
many interrelated statewide planning and accountability processes. 
While none will come to a complete halt without mission 
refinement, greater attention to mission will reduce confusion and 
Hill promote clearer articulation among processes. It is also 
important to note that each institution's selection of appropriate 
performance dimensions in the accountability process to some 
extent refines its conception of mission. 

This task is important because of the critical place of mission 
statements in such state-level processes as program review and 
resource allocation, euid because it provides a clear statement of 
accountability at the system level. It is important here to state 
what the system is to do, and what its particular assets are to 
look like. This responsibility encompasses the roles and missions 
of both public and private Institutions. Because the Commission 
has no authority to state n'.ssions for private institutions, their 
role in defining missions fqr public institutions becomes, from a 
systemic perspective, even uiore important. 

That the Commission develop ^ assessment mechanism for monitoring 
the performance of the highe education system in the state. 

As institutions are accountabl-^ for progress toward institutional 
objectives, so should the Cr.'- ^ssion assume an obligation to move 
Colorado higher ^c^'ocation b a whole toward system-level 
objectives. A Porolla^T r-- Lgation is to periodically assess and 
report on its conditic.A ^lu performance. 

Specific requirements for collectJjig and monitoring information 
will obviously vi^ry with the oi);5ectives established. Regardless 
of which are selected, however, some dimensions can be relatively 
easily monitored. Indeed, many can rely on information already 
available or obtained periodically from institutions through 
established reporting requirements. Access of students to the 
higher ediocation system^ for example, is relatively easy to 
determine through existing information. Others such as "quality" 
will be considerably more difficult. Adopting a definition of 
"quality" that emphasizee institutional effectiveness helps 
clarify the problem, but putri greater weight on the prior 
recommendation to refine institutional mission. 

In any event, specifying the meaiu( o.T assessment and the 
indicators to be used will in itself help clarify systemwide 
objectives^ Indeed, the process of identifying assessment 
procedures may work hand-in-hfiuid with identifying priority 
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statewide objectives. On occasion, the Commission may find it 
easier to begin with the question of what kinds of evidence it 
needs for accountability to help discover and articulate its own 
priorities. In the process, the Commission may also discover 
better ways to interpret and communicate current goals such as 
access, diversity, quality, and economic development. 

That the Commission establish a format for an annual systemwide 
accountability report to the Governor and General Assembly, and to 
the public, containing a) indicators of the condition and 
performance of the state system along priority dimensions, and b) 
information about the actions taken by the Commission to enhance 
systemwide effectiveness. 

Because accountability requires both assessment and communication 
of results, it is incumbent on the Commission to regularly report 
on systemwide performance. At the same time, the format(s) used 
should exemplify good practice and should implicitly set a 
standard for institutional reporting. 

Furthermore, because many readers of accountability reports will 
be uTiScquainted with higher education, communication formats must 
be as straightforward and clear as possible. Given this 
requirement, we recommend development of appropriate "performance 
indicators." Following guidelines established by the National 
Center for Education Statistics, an indicator set should: 

• consist of statistically valid information related to 
significant aspects of the educational system and can be a 
single valued statistic or a composite index; 

• provide a benchmark for measuring progress or regression over 
time, or differences across geographical areas or institutions 
at one point in time, such that substantive inferences oan be 
drawn from presentation of the data; 

• be representative of policy issues or aspects of education that 
might be altered by policy decisions; 

• be easily understood by a broad array of citizens concerned 
with education; 

• be based on relatively reliable data and not subject to 
significcuit modification as the response errors or changes in 
the personnel generating it. 

Furthermore, because the Commission, as an agency of state 
government, is required to annually report its actions, inclusion 
in the accountability report of a section describing steps taken 
to enhance system effectiveness would also be appropriate. 

That the Commission attempt to explicitly link the, various 
regulation and coordination processes it currently administers and 
utilize them as levers to effect change. 
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Any accountability program should be directly linked to each of 
the other major processes administered by the Commission. To the 
extent these processes are also directed toward enhancing system 
effectiveness, they will contribute to the understanding and 
achievement of common purposes. To the extent that such linkages 
are not explicitly recognized, the utility of these other 
processes as tools for promoting the achievement of system goals 
is diminished. Many implications can be identified: criteria for 
program approval and discontinuance should reference contributions 
to particular aspects of system effectiveness on the one hand, and 
reinforcement of institutional mission on the other; and resource 
allocation mechanisms should be assessed in light of the degree to 
which they create incentives or disincentives for accomplishing 
identified system goals. 

B» Recommendations to CCHE Regarding Institutional Action on Selected 
Statewide Priorities 

Recommendations that apply to the Commission's special responsibility to 
ensure institutional action and performance in particular identified areas 
of statewide priority are as follows: 

6. That the Commission specify as early as possible any areas in 

which all institutions will be expected to provide accountability 
information. 

Regardless of assigned mission and role, there are some system 
goals toward which all institutions are expected to contribute. 
Some of these goals are relatively straightforward—for example 
access to higher education for identified minority groups. Others 
are of common importeuice but are less easy to assess. Among these 
are general knowledge outcomes that all baccalaureate graduates 
should have, regardless of institution attended (for example, 
knowledge of broad areas of inquiry and investigation in the 
sciences, social sciences, humanities and the arts). Also among 
them are general skills outcomes associated with the statewide 
goal of an educated citizenry (reading, writing, speaking and 
computation) or "higher order" skills important for state manpower 
development (for example, analysis, synthesis, critical thinking, 
and problem-* solving) . 

In fairness to the institutions, the Commission must specify these 
areas fairly early in the process. Because such identified 
dimensions will constitute a core component of the assessment 
plans designed by each institution for accountability, a list of 
priority dimensions should be available promptly in order not to 
impede the local development process. 

7. That the Commission initiate a discussion with the governing 

boards and institutions to determine the most appropriate methods 
for assessing institutional performance on any dimensions 
identified in 6. above. 
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At least initially, we recommend that identification of common 
performance dimensions proceed without mandating a common method 
of assessment. At the same time, the Commission should begin 
discussions with the governing boards and institutions regarding 
the kinds of methods that institutions might consider for 
providing perform: ice information on each identified dimension. 
Out of such discussion, some common methods or approaches might 
wnerge. Alternatively, groups of institutions might undertake to 
develop common apprcache-s for their own purposes. In probable 
areas of statewide concern such as general knowledge and skills, 
moreover, more than one assessment method should be allowed, and 
institutions should be encouraged to use multiple methods. 

C- Recommendations to CCHE Regarding Institutional Accountability 
Processes 

Recommendations that apply to the Commission's role in promoting and 
monitoring the development of sound accountability programs on each of the 
campuses are as follows: 

8. That the Commission provide training seminars and technical 
assistance to institutional administrators with particular 
emphasis on a) selecting performance dimensions appropriate to 
each institution and b) identifying appropriate assessment 
approaches that might be used to gather evidence around each 
dimension selected. 

• 

On most campuses, outcomes^oriented accountability is a relatively 
new idea. As a resxilt, in attempting to respond to mandates to 
assess outcomes, institutions often lack direction and try to do 
more than is appropriate or possible. A set of seminars intended 
to define accountability in terms of institutional effectiveness, 
and covering available assessment alternatives and approaches in 
some detail would be a useful starting point. In addition to 
educating administrators on accountability issues, these seminars 
would provide a forum in which Commission staff can begin to 
develop guidelines and shape expectations about institutional 
accountability programs. At least as important, the seminars will 
help institutions to develop assessment processes geared toward 
internal needs for improvement. Substantial national experience 
with assessment has shown that institutions develop the best 
processes for accountability when these processes also have 
identifiable internal benefits. Because priority-setting and 
assessment approaches will be similar for institutions of similar 
kinds (for example, research taniversities, large community 
colleges, etc.)» parallel seminar presentations should be 
considered that involve representatives from similar institutions. 

As a follow-up to these seminars, there will likely be a need for 
on-campus assistance at some institutions. Development of 
effective institutional programs would also proceed more smoothly 
if the Commission could provide for direct technical assistance 
and could help defray the associated costs. 
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9« That the Commission develop a set of guidelines for developing 
institutional accountability plans, and request institutions to 
file such plans at an agreed-upon date. The guidelines should 
include the kiixJs of information about the local process that will 
be periodically required to allow Commission monitoring and 
statewide reporting. In filing a plan, each institution should 
describe a) particular performance dimensions selected for 
attention, b) proposed assessment methods, and c) communication 
plans. 

Submitted plans should be brief. Their intent would be to ensure 
general agreement on substance and form between CCHE and each 
institution before institutions invest heavily in implementing a 
complex assessment program. The mutual understanding created 
would help avoid false starts and would curb development of 
non-conforming programs at an early point. Furthermore, 
developing the guidelines for institutional plan submission 
creates a requirement for CCHE to state its expectations clearly. 

B. Recommendations Concerning Institution-Level Action 

Because this paper is centered on the state perspective, most of our 
attention has been directed toward recommending Commission action. 
Following our logic, however, several additional recommendations apply to 
governing boards and institutions. Listed briefly, these recommendations 
include the following: 

10. Each institution should take the initiative in critically 
reviewing and refining its own current statement of mission. 
Governing boards should encourage institutions to begin this 
process promptly, and should ensure that it takes place. 

In addition to a straightforward demand for clarity, the 
development of mission statements should reflect two additional 
principles. First, appropriate mission statements should include 
specification of outcomes as well as describing such traditional 
elements as breadth and depth of programs, quality of inputs, 
service region, and principal clientele. At the same time, the 
development of refined mission statements should be guided by a 
need to identify the institution's particular role in the 
statewide system of higher education and how this role 
differentiates it from other .institutions in the system. 

11. Using mission statements as a guide, each institution should 
identify specific performance dimensions for which it will hold 
itself accountable. This selection should recognize dimensions 
that are directly responsive to state priorities as well as those 
that are unique to the institution. 

Institutions are cautioned only to identify institution- wide 
dimensions at this juncture. Outcomes that are specific to 
particular programs should be treated in the program review 
process, not in the institutional accountability process. By 
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carefully identifying only institution-wide performance 
dimensions, the list can be kept relatively short. 

12. Each institution should develop an assessment plan and should 
review it with Commission staff before submission for approval. 

This step will help minimize misunderstanding once the program is 
inplemented. It may also promote learning on both sides about 
more effective ways to assess particular dimensions of 
performance. It is important for institutions to recognize that 
the Commission will be basing approval of institutional 
assessment plans on at least two criteria. First, it is expected 
that the proposed plan follows the Commission guidelines for 
institutional plans with respect to coverage, methods, and 
communication of results. Secondly, the plan should identify 
priority outcomes for assessment that are consistent with the 
institution's assigned mission, role, and scope. 

13 • Each institution should communicate the results of its assessment 
to appropriate audiences on a regular basis. 

Reporting should include actions taken as well as actual results 
of the process. It should also utilize formats tailored to the 
specific information needs of different institutional 
constituencies, for example, prospective students and their 
parents, local employers and governments, and citizens of their 
particular service region. 

14- Each institution should establish linkages between assessment and 
current institutional program review processes. 

In particular, institutional administrators should identify any 
specific programmatic outcomes that contribute directly to 
institutional or systemwide objectives and should ensure that 
program performance is assessed in these areas. Institutional 
administrators should also ensure that program outcomes are 
assessed in the review process and that these outcomes are in 
areas consistent with institutional role and mission. 
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APPENDIX A 



This Appendix is included to provide some general direction about the kinds of 
methods available for assessing particular effectiveness dimensions covered in 
this document. It is no^ intended as a full treatment of the topic; its 
primary purpose is to indicate the range of choices involved in data collection 
and to suggest most common current practice. 

1 . State/System Level Dimensions of Effectiveness 

a. General Education Levels/Population Literacy ; Information about 
the General Education Levels of the Population can be acquired in 
two distinctly different ways. The first and easiest is to use 
Census data on education levels of individuals in the population. 
Since these data are compiled by geographic region, the relative 
education levels of people in various regions can be readily 
determined. The most useful breaks are those that indicate the 
proportion of popxilations age 25 and over that have 1) less than a 
high school education and 2) less than an 8th grade education. 
The second alternative is implementation of a research program 
such as the National Assessment for Educational Progress Adult 
Literacy Exeun. Use of this approach would entail considerable 
expenditure, but it would also measure adult literacy directly 
rather than through surrogate measures such as years in school. 

b. EiBployment Characteristics ; In this arena it woxild be useful to 
compile information on number of employees categorized by both 
industry and occupation as well as information about employment by 
size of employing firm. 

c. Economic Conditions ; Indicators of economic conditions in the 
state that may be of interest are trends in the state gross 
product, trends in both personal and disposable income for the 
population of the state, and trends in numbers of business 
formations and failures. These are but examples of indicators in 
this arena. There are undoubtedly many others. 

d. Access/Participation ; Levels of student access or participation 
are typically measured in two ways. For full-time students 
(typically traditional college-age students) the measure of access 
is typically the ratio of entering college students to the number 
of high school graduates from the previous year. In Colorado, 
access by geographic area and by minority status are likely to be 
focal characteristics, although access on the part of students 
from different socioeconomic backgrounds is also at issue. The 
second approach to assessing access pertains more to part-time 
stxidents. This method involves comparing the attending student 
population with characteristics of th^ att^te population in 
relevant age cohorts (typically ages 25 to U)- 

e. Choice; While appropriate indicators of access or participation 
are measured relative to the system as a whole, measures of choice 
utilize the same student characteristics and review their 
distribution across institutions. Similar distribution analyses 
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can apply to any programs of particular state interest such as 
teacher education and medicine. Here the question is not whether 
the student gains access to the system but has appropriate 
opportunity for access to each part of the system. 

Minority Student Success ; A common indicator is tlie number of 
degi; ves gr&nted to minority students relative to institutional 
enrollments and to the population base in the relevant cohorts. 
Here the question is one of the relative performance of minorities 
ccMipared to majority students. 

Diversity ; The most common measures of diversity are the 
distributions of total enrollments, first- time students, and 
degrees granted across various sectors of the higher education 
system in the state. It will probably be most useful to show 
diversity in terms of public, independent and proprietary 
institutions, and to track the "market shares" of enrollments by 
major type of institution within these sectors. 

Institutional Quality ; This is the single hardest dimension for 
which to develop straightforward quantitative measures. It is 
made doubly difficul.t by the necessity of not defining quality 
solely in terms of the characteristics of major research 
universities. If quality is seen as effectiveness, and defined as 
producing outcomes called for in its mission and possessing the 
resources appropriate to its mission, quality in each institution 
is revealed by the extent to which it conforms to the "ideal" 
institution of its type. The scoring system utilised in Termessee 
to determine the extent to which each institution accomplishes its 
desired objectives represents one model for calculating "quality" 
in this format. 

Production of Trained Manpower ; A common measure is the number of 
degrees granted (by level) in each of the fields selected as being 
of primary importance to the rotate. A secondary measure is the 
proportion of graduates ia these fields placed witWJi the state of 
Colorado— data that can be acquired through use of a st-ident 
follow-up survey. Placement rates in selected fields will 
probably be important enough to be collected by all in^^tltutions 
in the state. 

Levels of Research ; The simplest indicators of research effort 
are based on expenditure levels for research. It is particularly 
useful to determine total expenditure levels by discipline and to 
subsequently determine the share of national expenditures on 
research in that discipline being conducted within the state. 
This represents a simple measure of the relative strength of the 
Colorado research enterprise vis-a-vis the rest of the nation. To 
the extent that applied research is a state priority, the 
proportion of research dollars coming to the institutions from the 
corporate sector provides some measxire of this phenomenon. 

Provision of Services : Assessment measures for this dimension are 
completely determined by the selection of those services 
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identified as being state priorities. If, for example, health 
care for the indigent is identified as a priority, a simple covuit 
of the nifflaber of individuals served provides one measure. A more 
sensitive measure would be the number served relative to the 
population living below the poverty level, as compared to the same 
proportion developed for those above the poverty level. 

1. Efficiency ; The concept of the efficiency of a system of higher 
education has received relatively little attention. Much more 
attention has historically bean devoted to institutional 
efficiency. Since the objective is to determine the efficiency 
with which the institution produces the array of outcomes implied 
by their mission, the traditional measure has been the cost of 
operating an institution relative tc the cost of operating other 
institutions (either within the state or elsewhere) having similar 
missions e The simplest and most frequently used measure of 
efficiency is institutional cost per FTE student. While simplest 
to calculate, tnis measure is also most subject to 
misinterpretation. The fundamental problem is that institutions 
within a system offer different programs, i.e., they differ in 
their emphasis on instruction, research, and public service. 
Within instruction they differ in emphasis by program and by level 
of instruction. These differences make comparability problematic 
to say the least, which in turn makes it difficult to svm across 
the performance of the institutions in such a way as to generate a 
valid measure of system output. Within an area such as 
instruction, it might be possible to adjust each institution's 
credit hour production to account for these differences and 
thereby develop common measures that could be summed across 
institutions. For exeunple, all credit hours might be expressed in 
terms of a standard lower division credit hour. Combined with 
expenditures for instzruction, the ingredients would be present for 
a rough indicator of efficiency: system performance could be 
evaluated over time, and possibly even again^st systems in other 
states if available data would support the required 
standardization routines. 

Institutional Level Dimensions of Effectiveness 

a. General Knowledge ; A number of methods are currently in use to 
assess student cognitive development—whether in general education 
or in particular disciplines. They include (l) standardized 
testing — for example ACT Assessment, Graduate Record Examination 
General or Field Exams, CLEP subject examinations, and various 
field-specific gradxiate admissions or certification excuninations 
such as the GMAT, LSAT, Engineering Readiness Test, etc.; (2) 
locally-designed assessment instruments constructed by discipline 
faculty~for example a senior comprehensive examination or 
capstone project, and (3) student surveys including self- report 
items on gains in cognitive knowledge — either commercially 
available (for example CIRP, ACT/ESS, SO IS, CSEQ) or locally 
designed. 
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General Skills ; Basically the same methods are available for 
assessing general (collegiate) skills. Among standardized tests, 
ACT/COMP is popular as it is intended to assess broad application 
of "liberal learning" skills. For assessing basic collegiate 
skills such as reading, writing, and computation, several state 
tests have been developed — for example the Florida CLAST and the 
New Jersey Test of Basic Skills. Many colleges use 
locally-designf 1 placement tests in reading, writing and 
computation that can also be used as post- test apsessment 
instruments. For sucli higher order skills as "critical thinking" 
or problem- solving, standardized tests or assessment methods crawn 
from psychology c^.n be used— for example the Kohlberg scale of 
ethical/moral development, or Perry's scheme for assessing 
critical thinking. Finally, self-reported skills may be usefully 
gathered by survey. 

Values/ Attitudes : Most institutions gather information of this 
kind by survey— either of currently enrolled or former students. 
All commercially-available student surveys contain such items — for 
example the Cooperative Institutional Research Program (CIRP), the 
ACT Evaluation Survey Service, the NCHEMS/ College Board Student 
Outcomes Information System (SOIS), or the Pace College Student 
Experience Questionnaire (CSEQ). Locally-^designed student surveys 
can be equally effective. If student populations are small, 
excellent use can also be made of "focus group" and individual 
interviewing techniques. 

Student Behavior While Enrolled ; Various important types of 
student behavior can be effectively documented using registration 
record systems. These Include term- to- term retention, program 
completion within a defined time period, student enrollment by 
major program, course- taking patterns, and course-completion 
rates. In order to generate appropriate liidicators, however, 
longitudinal record systems enabling students to be tracked 
throughout their enrollment must be constructed. Such "cohort 
studies" can be accomplished with varying degrees of 
sophistication ranging from simple systems operating on a 
microcomputer to complex mainframe record systems. 

Job Placement and Tenure ; Employment follow-up is usually 
accomplished through questionnaire surveys of former students. 
Generally, it is important to obtain Information from both program, 
completers and non-completers, as the latter may nevertheless have 
attained their occupational goals. Placement office records are 
also of value, but generally cover only those who have used 
placement serviC5^s. Surveys of employers, though Intended to 
ensure that cxirriculum content Is appropriate given employer 
neeas, will also often contain Items on former student placement 
and tenure. 

Subsequent Education ; Like job placement, the primary method for 
obtaining Information is former student surveys. Generally 
important are (1) the Institution(s) attended, (2) programs 
enrolled for, (3) success iti transferring credit, (4) subsequent 
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academic performance including completion and grade-point, and (5) 
assessment of the relevemce and quality of prior instruction 
received given subsequent education* Where clear linkages among 
institutions are present, for example an eurticulation agreement or 
a history of substantial transfer, agreements to share student 
records can be negotiated. Alternatively, state or system 
authorities in many states periodically conduct transfer studies 
that track students from institution CrC institution.* A final, 
rarer, alternative is to construct a "unit record system" for the 
entire state or system in which student records for all 
institutions are maintained centrally in a common format. 



g. Certification, Licensure, and Professional Contributions ; 

Professional career development is also most commonly obtained by 
surveys of former students tt>at include certification, licensure, 
professional memberships and recognitions, etc. 
Certification/licensure records are also directly obtainable in 
many cases. Where the institution administers a licensing 
examination, such records are immediately at hand. In cases such 
as Nursing and Accounting, moreover, testing is visible and 
periodic, and institutions can obtain passage results from the 
administering agency. Where certification/licensing is 
administered by the state, it is possible to match certification 
records with student records to produce an index. 



h. Graduates Remaining In^State ; Follow-up surveys commonly report 
state of residence, although former student records may be 
unreliably maintained after more than about five years. 

i. Research and Scholarship ; Sponsored or assigned research is 
generally documented by dollars awarded disaggregated by 
discipline or type of activity. Another traditional indicator of 
formal research productivity is the number of juried publications 
or citations indices. Unsponsored research or service is more 
difficult to document. Generally such outcomes emerge from 
faculty activity surveys periodically administered, or from 
constituency surveys of employers, local governments, civic or 
volunteer organizations, etc* for purposes of ascertaining general 
community impact. 

j. Community/state Service ; Because "service" entails so many 

things, a number of data gathering methods are commonly practiced. 
.Service contributions of faculty, staff or students can be made a 
part of surveys of these populations generally imdertaken for 
other purposes. These would include both volwtary and assigned 
activities* Use of campus facilities by the community and such 
things as citizen participation/attendance at institutional events 
may be doci'mented by institutionally-maintained records. Finally, 
community impact surveys (generally mailed or telephone) of 
citizens, and needs assessment studies (mailed or telephone 
sxorveys or in«»depth interviews) of area businesses, governments, 
and civic or volunteer organizations are often used to document 
service contributions. 
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Many of the above methods can be effectively combined. For example, 
almost all can be directly or indirectly assessed by means of a 
well-conceived former student siirvey, although knowledge and skill 
outcomes are in this case documented only by self- report. Moreover, a 
number of instruments (the ACT/COMP for example) attempt to ascertain both 
general knowledge and general skills outcomes. As in any other 
"indicators** approach, it is important to develop and use more than one 
method or approach. Furthermore, it is important to interpret results as 
a "profile" of performance, rather than examining each indicator 
independently or viewing each as a minimum standard to be met. Finally, 
as emphasized throughout, the investment in data gathering should match 
the priority of the outcome in question, given the institution's assigned 
mission, predominant program direction, and primary clientele. 
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APPENDIX B 



In the course of preparing, this paper, we consulted n merous references on 
state higher education policy, assessment methods anc ^^pproaches, and on the 
statutory and political context for Colorado higher e vocation. Because the 
paper is intended for policy guidance, we chose a forrat that minimized direct 
citations. As the Commission moves to implement an accountability program, 
however, it will need furU^er guidance on the details of alternative approaches 
and on the experiences of other states and institutions with similar types of 
programs. The intent of this brief Appendix is to provide a short list of 
references for guidance in developing implementation alternatives. References 
are provided in the areas of (1) existing relevant Colorado statutes or public 
planning documents, (2) approaches and methods for assessing institutional 
effectiveness and educational outcomes, (3) the impact of assessment and 
evalxjation infonnation on institutions and state systems, and (4) appropriate 
state and institutional planning and policymaking. 

1. References on the Context of Colorado Higher Education: 

General Assembly of the State of Colorado. House Bill No> 1187 > Denver, 
Colo.: May 1985. 

Colorado Commission on Higher Education. Policy and General Procedures: 
Quality Incentive Program . Denver, Colo.: 1986. 

Colorado Commission on Higher Education. Policy and General Procedures for 
State Level Review of Existing Programs . Denver, Colo. 

Colorado Commission on Higher Education. Admission Standards Policy of the 
Colorado Commission on Higher Education . Denver, Colo.: October 1986. 

Colorado Commission on Higher Education. "Goals and Objectives for the 
System of Postsecondary Education." Excerpts from the Master Plan. 
Denver, Colo. 

Colorado Commission on Higher Education. Summary Notes from 1986 Community 
Roundtables. Denver, Colo., 1986. 

Colorado Commission on Higher Education. "Memorandum to Commissioners for 
Master Plan Retreat." Denver, Colo.: January 28, 1937. 

Colorado Commission on Higher Education. Laws of the Colorado Department 
of Higher Education . Denver, Colo.: February 1986. 

2. References on Assessment Approaches and Methods: 

Useful general discussions of assessment methods and approaches include: 

Adelman, Clifford (Ed*). Assessment in American Higher Education; Issues 
and Contexts . Washington, D.C.: Office of Educational Research and 
Improvement, U.S. Dept. of Education, 1986. 

Alyemo College Faculty, Assessment at Alvemo College . Milwaukee, Wise: 
Alvemo Productions, 1985. 
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As tin, Alexander W. Four Critical Years: Effects of College on Beliefs, 
Attitudes and Knowledge > San Francisco: Jossey-Bass, 1977. 



Bowen, Howard R. Investment in Learning; The Individual and Social Value 
of American Higher Education > San Francisco: Jossey-Bass, 1977. 

Ewell, Peter T. (Ed.). Assessing Educational Outcomes . New Directions for 
Institutional Research #47. San Francisco: Jossey-Bass, 1985. 

Ewell, Peter T. The Self-Regarding Institution: Information for 
Excellence . Boulder, Colo.: National Center for Higher Education 
Management Systems [NCHEMS] , 1984. 

Ewell, Peter T. Information on Student Outcomes: How to Get It and How to 
Use It . Boulder, Colo.: NCHEMS, 1983. 

Feldman, Kenneth A. and Theodore M. Newcomb. The Impact of College on 
Students . San Francisco: Jossey-Bass, 1969. 

Jones, Dennis P. "Indicators of the Condition of Higher Education." Paper 
prepared for the National Center for Educational Statistics. Boulder, 
Colo.: NCHEMS, October, 1985. 

McDonald, Jean G. "College Quality: Measuring What Students Learn in Higher 
Education." In Time for Results , The Governors' 1991 Report on 
Education— Task Force on College Quality Supporting Works. Washington 
D.C.: The National Governors' Association, Axigust, 1986. 

Micek, Sidney S., Allan L. Service, and Yong S. Lee. Outcome Measures and 
Procedures Manual . Field Review Edition. Boulder, Colo.: NCHEMS, 1975. 

Pace, C. Robert. Measuring the Outcomes of College . San Francisco: 
Jossey-Bass, 1979"! 



Useful tasic references on conducting particular kinds of assessment and 
outcomes studies include the following: 

Armijo, J. Frank, Sidney S. Micek, and Edward M. Cooper. Conducting 

Community Impact Studies: A Handbook for Community Colleges . Boulder, 
Colo,: NCHEMS, 1978. 

Caffrey^ John and Herbert H. Isaacs. Estimating the Impact of a College or 
University on the Local Economy . Washington, D.C.: American Council on 
Education, 1971 . 

California Community Colleges. FIPSE Project Item Bank . Sacramento: 
Chancellor's Office, California Community Colleges, 1983. 

Dillman, Don A. Mail and Telephone Surveys: The Total Design Method . New 
York! John Wiley, 1982. 

Ewell, Peter T. Student Outcomes Questionnaires: An Implementation 
Handbook , Second Edition, Boulder, Colo.: NCHEMS, 1983. 
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Pace, C. Robert. Higher Edxicatlon Measurement and Evaluation KIT . Los 
Angeles: Center for the Study of Evaluation, University of California at 
Los Angeles, 1971 • 

Terenzini, Patrick T. "Designing Attrition Studies." In Ernest T. 
Pascarella (Ed.), Studying Student Attrition . New Directions for 
Institutional Research #36. San Francisco: Jossey-Bass, 1962. 



References on Institutional Impact of Assessment 

A number of institution- level assessaient programs have been thoroughly 
documented with respect to both their impact on the institution and their 
implementation. These studies are useful to discover alternative ways of 
approaching assessment and evaluation and to discover the benefits of the 
process for institutions themselves; 

Banta,. Trudy W. (Ed.). Performance Funding in Higher Education; A Caie 
Study . Boulder, Colo*: NCHEMS, 1985- 

McClain, Charles J. Degrees with Integrity: A Value-> Added Approach to 
Undergraduate Assessment . Washington, D.C.: American Association of 
State Colleges and Universities, 1984. 

Mentkowskl, Harcia and Austin Doherty. Careering After Colle«:e: 

Establishing the Validity of Abilities Learned in College for Later 
Careering and Professional Performance . Milwaukee Vfisc.s Alvemo 
Productions, 1983. 



References on State and Institutional Higher Education Policy 

Some useful references on policy tools for effecting improvement in state 
higher education systems and for improving institutional mission , planning 
and evaluation procedures are as follows: 

Caruthers, J. Kent and Gary B. Lott. Mission Review: Foundation for 
Strategic Planning . Boulder, Colo.T NCHEMS, 1981. 

Education Commission of the States. Transforming the State Role in . 

Undergraduate Education: Time for A Different View . The Report of the 
Working Party on Effective State Action to Improve Undergraduate 
Education. Denver, Colo.: ECS, 1986. 

Ewell, Peter T. Levers for Change > The Role of State Government in 
Improving the Quality of Postsecondary Education . Denver, Colo.: 
Education Commission of the States, 1985. 

Ewell, Peter T. The State Role in Assessing College Outcomes: Policy 

Choices and Probable Impacts . Wr^hington, D.C.: The National Governors' 
Association, June, 1986. 

National Govemors' Association. Time for Results: The Governors' 1991 
Report on Education . Washington, D»C.: The National Governors' 
Association Canter for Policy Research and Analysis, 1936 
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.Jones, Dennis P. Higher Education Budgeting at the State Level; Concepts 
and Principles ^ Boulder, Colo.: NCHEMS, 1984. 
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